Data science became one of the foundations of contemporary technology in the last ten years by cutting through the clutter. Today, practically every new product, business venture, and industry-leading organization is powered by the capacity to anticipate the future based on prior data.

The adoption of predictive models—algorithms capable of 'fitting' (learning) past data—brought to the introduction of these capabilities. Therefore, it should come as no surprise that much effort has gone into automating, simplifying, and democratizing the processes of choosing and refining models.

The main issue with data science

The proper data must be fed to the algorithms, not the algorithms themselves.

(Again, not necessarily true for computer vision/audio and NLP use cases.) Nowadays, very few people are developing novel algorithms when applying data science to commercial challenges. We have been using the same models for over ten years. The 1960s saw the development of SVMs, the preceding century saw the introduction of gradient boosted trees (XGBoost), and linear regression was presumably invented about 1000 BC.

Like any other profession that has grown out of new technology and had an impact on society, data science is not unique. Each new development is more thrilling than the one before it and opens up many avenues to difficult problems and challenges. We nevertheless keep applying the same models.

Automating data discovery and feature generation

I think that resolving the data component of the model-building process will enable countless opportunities for the businesses and goods around us. Currently, this is the primary barrier for all kinds of enterprises.

A large, forward-thinking organization that tries to extend data science across several operations and use cases will quickly discover that there aren't enough data scientists that can navigate various features and data sources.

The surprisingly high amount of manual work required to examine a new source (hypothesizing, matching and integration, aggregating and generation of features) only to see if it actually enhances a predictive model would likely be a challenge for a corporation wanting to acquire relevant data sources to enhance existing models. If a business doesn't own an infinite supply of data, it will have to work harder and harder to acquire better data and produce high-value features.

Automation of data source exploration and feature generation in the context of model construction can address each of these challenges.

As automation in data science increases, everyone is asking whether data scientists will eventually be replaced. No.

Data scientists will always be needed to turn a business problem into a 'data problem,' to determine the proper thing to predict, to specify the search space for the problem, to prevent data leakage, and to validate the results to make sure they make sense and can be utilized in production to have an influence on the business.

It's time to start concentrating on the data science component.